The present invention relates to augmented reality and, in particular, it concerns a system and method providing network-based real time augmented reality for mobile devices.
Augmented reality (AR) refers to a manner of presenting supplementary information or graphic content overlying and visually combined with a view of the real world, whether viewed directly or as images on a display. In the case of a directly viewed scene, the supplementary information is generally displayed via a transparent display arrangement. In the case of images viewed on a display, the supplementary content is generally combined on the same display.
A key issue in implementation of any AR solution is the resolution of registration achieved between the real world scene and the supplementary information. For low resolution AR, for example, where it is sufficient to provide a simple text label deployed in the general direction of features occupying a large part of the field of view, it is possible to implement a rudimentary AR system using GPS together with an inertial navigation system (INS) or another basic position sensor suite. An example of such an implementation is described in “Shared Database of Annotation Information for Wearable Augmented Reality System”, Koji Makita et al., Graduate School of Information Science, Nara Institute of Science and Technology, 8916-5 Takayama, Ikoma, Nara, 630-0192 Japan, pp. 464-471 (XP-002408355). A range of commercial applications have recently become available for various smart-phone devices which provide AR functionality based on built-in position sensors. The performance of such applications is limited by the precision of the position sensors and the noise level in their readings.
In more demanding AR applications, it may be necessary to provide more precise registration of the supplementary information with the real world view than can be achieved using low-cost position sensors, for example, allowing the system to distinguish between two objects which appear small and close together in a sampled image. In order to achieve high resolution registration of the supplementary information to the real world, techniques have been developed based on image processing through which sampled images are registered to a geographic information system (GIS) including ortho-photo data. Examples of this technology may be found in R. P. Wildes et al., “Video georegistration: algorithm and quantitative evaluation”, ICCV 2001, Vol. 2, pp. 343-350; and in R. W. Cannata et al., “Autonomous video registration using sensor model parameter adjustments”, IEEE Workshop on Video Registration (with ICCV 2001).
Image-processing-based AR applications such as those described in these articles potentially offer accurate pixel-by-pixel registration to the geographic database, and hence provide precise positioning of supplementary AR information sufficient to identify all visible features. However, this approach requires access to a large database of reference images, and strong processing capabilities. In practical terms, the hardware requirements for implementing image-processing-based AR have until now been generally prohibitive for domestic consumer electronics devices and other portable devices. Furthermore, remote image processing is typically not an effective option for mobile devices networked via a wide area network (WAN) due to the significant delay introduced by uploading images over the limited bandwidth of the WAN.
There is therefore a need for a system and method providing network-based real time augmented reality for mobile devices which would allow image-processing-based AR to be used effectively within the processing and communication bandwidth limitations of networked portable devices.